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Many states seek to increase the education levels of their early childhood education (ECE) workforce 
to improve the quality of care for children. Oregon encourages all ECE workforce members to sign up 
for a career lattice, a career pathway system that helps them determine goals related to increasing 
their education. The state also offers incentives for reaching specific steps in the career lattice and 
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trials in 2018 and 2019 to test whether sending emails and offering different financial incentives to 
Oregon ECE workforce members increased career lattice sign-up and increased education and training 
levels or workplace retention. The study found that sending emails encouraging career lattice sign-up 
had no detectable impact on career lattice sign-up or workplace retention. Sending emails offering a 
monetary incentive at an earlier-than-usual step on the career lattice had a positive impact on training 
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or workplace retention. Sending emails about automatic enrollment in a scholarship program had no 
detectable impact on scholarship use, career lattice movement, college credit hours earned, or workplace 
retention. Lastly, after participants were randomly assigned to study groups, the email campaigns were 
implemented as planned, reaching all intended participants, although the interventions ended sooner 
than planned because of a state policy change. The findings suggest that low-touch interventions such 
as emails have promise for increasing training hours but are not sufficient to induce changes in career 
lattice sign-up, continuing postsecondary education, or workplace retention for Oregon ECE workforce 
members. These results have implications for future research, in addition to demonstrating how better 
messaging and supports can mitigate barriers to further education and training and how email campaigns 
can be leveraged for workforce communication efforts. This information is particularly relevant for state 
agencies and education and training providers. 


Many states seek to improve the educational attainment of their early childhood education (ECE) workforce, and 
nearly all states require ongoing professional development’ for ECE professionals (Gomez et al., 2015). Compared 
with the K-12 teacher workforce, the ECE workforce has lower educational attainment levels. Only 46 percent of 
preschool teachers age 25 and older have at least a bachelor’s degree compared with 95 percent of elementary 
and middle school teachers (U.S. Department of Labor, 2018). While nearly all states have identified a core set of 
competencies and skills required for the ECE workforce, there is no consensus across states on how those com- 
petencies and skills translate into minimum education requirements or on the requirements for different types 
of workplaces (Whitebook et al., 2016). Higher education levels among ECE workforce members, particularly lead 
teachers, have been linked to improvement in classroom quality, but research has 
yet to show a clear link to child outcomes (Manning et al., 2017). However, some 
state policymakers hypothesize that increasing the education level and skills of For additional information, 
the ECE workforce (particularly workforce members with less than an associate including background 
degree) will result in higher quality care for children and thus better child out- on the study, technical 
comes (Manning et al., 2017). (See box 1 for key terms used in this report.) 


methods, and supporting 
analysis, access the 


ee report appendixes at 
1. Professional development is defined in this study as any training or education and includes both https://go.usa.gov/xFRyG. 
training and college coursework hours (see box 1 for definitions of key terms). 
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Box 1. Key terms 


Behavioral nudge. An encouragement to alter a person’s behavior that does not eliminate choice or significantly change eco- 
nomic incentives. A behavioral nudge changes the presentation of choices in such a way that people are more likely to choose one 
option than another (Thaler & Sunstein, 2008). An example of a nudge is a text message reminder to submit an application. In this 


study emails are used as behavioral nudges. 


Career lattice. A career pathway or ladder system that guides workforce members to higher education and training levels, often 
used by states for their early childhood education workforce (Karoly, 2012; LeMoine, 2008; Limardo et al., 2016). 


Control group. A group of study participants that does not receive any intervention or change in program or benefits and contin- 
ues in the status quo. In this study the control group was randomly assigned to not receive any emails regarding signing up for the 
career lattice or containing information on monetary incentives or scholarships and was not eligible for monetary incentives at an 
earlier-than-usual step in the career lattice or for automatic enrollment in a scholarship program. 


Early childhood education (ECE). Refers to all forms of care and education programs for children ages O-5, including infant/ 
toddler care, preschool, Head Start (including Early Head Start), home-based or family child care, and center-based care. 


Early childhood education workforce. Staff members (in teaching and nonteaching positions) employed in licensed care facil- 
ities (including infant/toddler care, preschool, Head Start, Early Head Start, home-based or family child care, and center-based 
care) that serve children ages birth to 5. The terms ECE workforce members, ECE staff members, and ECE professionals refer to the 


ECE workforce. 


Intent-to-treat and treatment-on-the-treated effects. Estimates of the impact of an intervention, change in program, or 
change in benefits that differs from the status quo, calculated through regression analysis. An intent-to-treat analysis measures 
impact on the treatment group regardless of whether the treatment group member received treatment (in this study, regardless 
of whether the person received the emails). A treatment-on-the-treated analysis measures impact only on those who received 
treatment; in this study, only on those who received the emails. 


Professional development. Any training or education available in the ECE field, including workshops, conferences, initial training, 
and courses for college credit that count toward a degree, as well as continuing education (following the definition in Whitebook 
et al., 2009). In this study professional development refers to both training and college coursework. 


Treatment group. A group of study participants randomly assigned to receive an intervention, change in program, or other bene- 
fits that differ from the status quo. In this study the treatment group received emails regarding signing up for the career lattice or 
containing information on monetary incentives or scholarships and was eligible for monetary incentives at an earlier-than-usual 
step in the career lattice or for automatic enrollment in a scholarship program. 


Workplace retention. Being employed at the same early childhood education workplace at both the start and end of the study period. 


Oregon offers many professional development opportunities for the ECE workforce. For example, 14 community 
colleges and 12 public and private universities offer ECE degree programs (Montoya et al., 2018). Training can 
include in-service sessions and courses on specific topics, all of which can be offered through a variety of organi- 
zations, such as state agencies, child care resource and referral programs, and independent trainers. 


Despite these options, ECE workforce members face systemic barriers to enrolling and investing in professional 
development that decrease participation, such as language and cultural differences, a need to balance work and 
family obligations, and low wages (Ackerman, 2004). Less than a third of the Oregon ECE workforce has at least a 
bachelor’s degree (Oregon Center for Career Development in Childhood Care and Education & Oregon Child Care 
Research Partnership, 2019). A majority of ECE professionals are women (94 percent). Close to 15 percent report a 
language other than English as their primary language, and Latinx professionals account for almost 20 percent of 
the state ECE workforce. Given the average educational attainment and demographic makeup of the state’s ECE 
workforce, systemic barriers to enrolling and investing in professional development might be especially relevant 
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in Oregon. Wages for ECE professionals are also low. Nationally, the 2016 median annual wage was $28,790 for 
preschool teachers ($13.84 median hourly wage; U.S. Department of Labor, 2017a) and $21,170 for other ECE work- 
force members ($10.18 median hourly wage; U.S. Department of Labor, 2017b). 


To increase the education and training level of the ECE workforce—and to reduce barriers to further education 
and professional develooment—states have adopted financial incentives to encourage individuals to move along 
a career lattice and participate in a workforce registry (Ackerman, 2004, 2016; Gomez et al., 2015). A career lattice 
is a career pathway system that helps workforce members set goals related to increasing their education. Career 
lattices for ECE workforce members have become a common feature of state professional development systems 
across the country (Karoly, 2012; LeMoine, 2008; Limardo et al., 2016). They are often combined with a workforce 
registry, which allows states to monitor and evaluate ECE workforce members while tracking education level, 
credentials, training experience, employment history, and position in the career lattice (Karoly, 2012). 


Oregon stakeholders would like to encourage ECE workforce members—especially those with less than an asso- 
ciate degree—to sign up for the state’s career lattice, Oregon Registry Steps, and reach higher education levels 
(including both college coursework and training). The state theorizes that workforce members who sign up might 
benefit from a greater sense of identification as an ECE professional. That is, if individuals see that they are part 
of a broader professional system, know where they stand in the career lattice, have a clear path to advance 
professionally, and understand the financial incentives tied to advancement, they might be more likely to pursue 
education and professional development. In addition, greater career lattice participation would give Oregon 
stakeholders a more complete understanding of workforce education levels, which could lead to more targeted 
policies. However, there is little research on how to encourage participation in a career lattice or on career lattice 
movement (see appendix A for a review of literature related to this study). This study examined one set of efforts 
to encourage career lattice participation and movement. 


The Oregon Center for Career Development in Childhood Care and Education (OCCD), based at Portland State Uni- 
versity, manages the registry system (called the Oregon Registry Online) that documents the licenses and profes- 
sional development of ECE workforce members and the career lattice. All of Oregon’s approximately 25,000 ECE 
workforce members, whether in licensed child care centers or home-based care centers, must meet certain train- 
ing requirements and submit information to the Oregon Registry Online to document compliance with require- 
ments.” Registering with the Oregon Registry Online is required, but participating in the career lattice is voluntary. 
Although Oregon provides a financial incentive for career lattice participation, as of 2018 only 42 percent of ECE 
workforce members in the state’s licensed child care centers and home-based care centers had signed up for the 
lattice and had their training and education translated into a step—a level between 1 and 12 that corresponds with 
a certain education threshold (Oregon Center for Career Development & Oregon Child Care Research Partnership, 
2019). The low participation rate could be due to lack of awareness or information about the career lattice. (See 
table B1 in appendix B for details on Oregon’s career lattice.) 


OCCD partnered with the Regional Educational Laboratory Northwest to conduct the current study, which exam- 
ined ways to provide an incentive, encourage participation in Oregon’s career lattice, and increase education 
levels. This study used two randomizations (a method for randomly assigning study participants to a treatment 
or control group) to test three interventions, all of which are behavioral nudges communicated through email: a 
behavioral nudge encouraging career lattice sign-up, financial incentives in the form of awards for certain educa- 
tion levels, and financial incentives in the form of automatic scholarship enrollment (figure 1; see also figure B3 


2. The training requirements vary by position and workplace type. For example, as of 2020 a teacher in a licensed child care center in 
Oregon is required to take at least 15 hours of training every year, and a home-based care provider is required to take 10 hours of 
training every two years. 
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Figure 1. The study used two separate randomizations in 2018 and examined three interventions 


Oregon early childhood education workforce 


In career lattice 


Not in career lattice 


Sign-up randomization Incentive randomization 


Emails offering 1-6 7-8 8.5-12 


career lattice sign-up ] l 
Intervention: Intervention: 


Emails offering Emails offering 


Intervention: At steps | At steps | | At steps | 


monetary incentives automatic enrollment 
tied to career lattice in scholarship 


Note: Interventions are designated by a darker outline around the box. Step 1 includes registering with the Office of Child Care, meeting Department 

of Human Services enhanced rate training requirements, or having 12 hours of training. Steps 2—5 include earning additional college quarter credits or 
training hours. Step 6 refers to having up to nine college quarter credits or 90 hours of training in specific core knowledge categories. Step 7 is attaining 
the Child Development Associate (CDA) credential, 12 quarter credits in two core knowledge categories, or 120 hours of training in specific core knowl- 
edge categories. Step 7.5 is attaining the CDA credential with eight college quarter credits, 20 quarter credits, or 200 hours of training in specific core 
knowledge categories. Step 8 is attaining an articulated certificate in the field, 30 quarter credits, or 300 hours of training in specific core knowledge 
categories. Step 8.5 is attaining an articulated certificate in the field, 40 quarter credits, or 400 hours of training in specific core knowledge categories. 
Steps 9-12 are attaining an associate degree or higher, 60 or more quarter credits, or 600 or more training hours in specific core knowledge categories. 


Source: Authors’ compilation. 


in appendix B). OCCD was also interested in understanding the impacts on workplace retention and workforce 
members’ identification as early learning professionals. 


Behavioral nudges have been shown to be effective in postsecondary education settings, while financial supports 
and incentives have shown promise among the ECE workforce. Behavioral nudges change the way information is 
presented so that people are more likely to choose one option over another (Thaler & Sunstein, 2008). Behavioral 
nudges can be sent, for example, through email or text message and have been shown to increase postsecondary 
education outcomes, such as college enrollment and persistence and financial aid take-up (Castleman & Page, 
2015; Page et al., 2020). Descriptive, noncausal studies have found that financial incentives of S80-S6,000 are 
associated with increases in ECE professionals’ education levels and participation in professional development 
(Park-Jadotte et al., 2002; Weber & Trauten, 2008; Whitebook & Bellm, 2004). In addition, one study of an 
Oregon scholarship program aimed at increasing ECE workforce members’ education levels and participation in 
professional development found that educators who received a scholarship were more likely to progress in the 
state’s career lattice and engage in more training hours than educators who did not receive a scholarship (Weber 
& Grobe, 2014). 


The first of the study’s two randomizations (the sign-up randomization) included ECE workforce members in the 
Oregon Registry Online who agreed to participate in the study and were not already registered in the career 
lattice. This part of the study examined one intervention: OCCD sent emails every three months to individuals 
who were randomly assigned to receive them (the treatment group). The emails described the benefits of the 
career lattice and encouraged recipients to participate. The control group did not receive any emails from OCCD 
but could sign up for the career lattice on OCCD’s website if they chose. 
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The second randomization (the incentive randomization) included ECE workforce members who agreed to par- 
ticipate in the study and were already registered in the career lattice at step 7, 7.5, or 8, all of which reflect an 
education level below an associate degree (step 9).2 OCCD observed that workforce members often reach steps 
7-8.5 but then do not continue professional development, essentially stalling in the lattice. As a result, OCCD 
hypothesized that additional incentives might be needed to encourage professional development beyond steps 
7-8.5. This part of the study examined two interventions and included two separate treatment groups and one 
control group. In the first intervention OCCD sent emails every three months to one treatment group announcing 
eligibility for monetary awards for individuals who progressed in the career lattice and encouraging enrollment in 
professional development. The monetary awards redistributed to steps 7.5-8.5 the $200 award typically available 
when an ECE workforce member earns an associate degree or reaches the equivalent level in community-based 
training hours (which occurs at step 9). This redistribution to an earlier-than-usual step in the career lattice did 
not add to the cost of the program for the state, making it easier to implement statewide if found to be effective. 
In the second intervention OCCD sent emails every three months to another treatment group notifying recipients 
that they had been automatically enrolled in a community college scholarship program, removing the need for a 
scholarship application. The emails for the automatic scholarship enrollment treatment group also included infor- 
mation about college registration deadlines and OCCD services available to help determine which courses to take, 
but the emails did not include explicit encouragement to continue college coursework and training. The control 
group did not receive any emails from OCCD but was eligible for the monetary incentive at the usual step (9) in 
the career lattice and could apply for the scholarship program through the OCCD website. 


In addition to testing the impact of the three interventions, the study examined the implementation of the three 
email campaigns to understand whether the interventions were implemented as planned (see appendix B for 
examples of the emails). The study did not test the impact of the career lattice, the typical monetary incentive 
program available to those in the career lattice, or the scholarship itself. Rather, it tested the impact of low-cost 
ways to encourage professional development. For both the sign-up and incentive randomizations the study also 
tested whether the interventions were related to greater identification as an early learning professional. That is, 
ECE workforce members might see themselves as part of a larger system and thus feel more connected to their 
profession if they receive more communication from OCCD, participate in the career lattice, and increase partic- 
ipation in professional development. The study also examined whether the interventions from both randomiza- 
tions impacted workplace retention, as ECE professionals might be more likely to stay in their current role if they 
feel supported by their workplace in their pursuit of professional development. For the incentive randomization 
the study tested whether sending the emails increased reported motivation to complete a degree or to engage in 
college coursework in the next year or the next five years. 


Because of a state policy change, the interventions ended four months earlier than planned. In August 2019, one 
year into study implementation, the Oregon Department of Education Early Learning Division notified ECE pro- 
fessionals statewide that the typical financial incentive award program based on career lattice steps would end 
and that they should apply for the awards before funding ran out. Reasoning that the state communication to the 
ECE workforce could lead some workforce members to disengage from OCCD communications, the study team 
decided to end the interventions before the announcement and moved up the timeline for the study’s follow-up 
survey. The fifth and final email was sent to participants in July 2019, four months earlier than originally planned. 
(The study originally intended to send the fifth email in August 2019 and a sixth and final email in November 2019.) 


3. The career lattice steps that were eligible for this randomization were step 7 (the Child Development Associate credential, 12 quarter 
credits, or 120 training hours), step 7.5 (the Child Development Associate credential and eight quarter credits, 20 quarter credits, 
or 200 training hours), and step 8 (an articulated certificate in the field, 30 quarter credits, or 300 training hours). Step 8.5 was not 
included in the randomization because that step was already eligible for a $200 incentive to achieve the next step. See table B1 in 
appendix B for more information. 
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The study findings provide information that can guide future research in using emails and monetary incentives to 
encourage the ECE workforce to participate in professional development. The findings shed light on how better 
messaging and supports can help mitigate barriers to both career lattice participation and further professional 
development, as well as how email campaigns can be leveraged for future studies. This information could be 
helpful to state policymakers and professional development providers as they develop programs to provide 
incentives for the ECE workforce to participate in professional development. 


Research questions 


This study addressed three research questions related to the interventions’ impact and one research question 
related to the interventions’ implementation. Study interventions were conducted between August 2018 and July 
2019. 


Impact research questions 


1. Among ECE workforce members not signed up for Oregon’s career lattice in 2018, did emails impact sign-up for 
the career lattice or workplace retention at the same workplace? 


a. Did the intervention impacts vary by demographic group, workplace role, workplace type, urban or rural 
location, or distance to the closest in-state public college or university? 


b. Did the intervention influence the secondary outcome of participants’ identification as an early learning 
professional? 


2. Among ECE workforce members at career lattice steps with at least the Child Development Associate credential 
and less than an associate degree or equivalent community-based training hours in 2018, did emails coupled 
with a monetary incentive impact the number of training hours recorded, the number of college credit hours 
earned, career lattice movement, or workplace retention? 


a. Did the intervention impacts vary by demographic group, workplace role, workplace type, urban or rural 
location, or distance to the closest in-state public college or university? 


b. Did the intervention influence the secondary outcomes of participants’ identification as an early learning 
professional, reported motivation to take college coursework in the next year or the next five years, or 
reported motivation to complete a credential, certificate, or degree? 


3. Among ECE workforce members at career lattice steps with at least the Child Development Associate credential 
and less than an associate degree or equivalent community-based training hours in 2018, did emails coupled 
with automatic enrollment in a scholarship program impact scholarship use, career lattice movement, the 
number of college credit hours earned, or workplace retention? 


a. Did the intervention impacts vary by demographic group, workplace role, workplace type, urban or rural 
location, or distance to the closest in-state public college or university? 


b. Did the intervention influence the secondary outcomes of participants’ identification as an early learning 


professional, reported motivation to take college coursework in the next year or the next five years, or 
reported motivation to complete a credential, certificate, or degree? 
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Implementation research question 


4. To what extent were the email campaigns for the career lattice sign-up and incentive interventions implement- 
ed as intended? 


a. To what extent did participants receive, open, and click on the emails? 


b. What were the implementation successes and challenges of the email campaigns as reported by OCCD staff 
and participants? 


The data sources, sample, analytic methods used for this study, and study limitations are presented in box 2. 
Additional information about the study data sources, sample, and methods is in appendix B. 


Box 2. Data sources, sample, methods, and limitations 


Data sources. This study used four data sources: 

e Oregon Center for Career Development in Childhood Care and Education (OCCD) administrative data from the Oregon Registry 
Online for September 2018—December 2019. These data included information on study participant workplace name, workplace 
address, license status, credits earned, career lattice step, incentives earned and received, scholarship use, work experience, 
and demographics (education level, race/ethnicity, primary language, and birthdate). The study’s main outcomes came from 
this data source. 

e OCCD survey data gathered from a baseline survey administered in July-August 2018 and a follow-up survey administered in 
September—October 2019. Topics included plans for future professional development, motivators and barriers to professional 
development, and how much participants identified as an early learning professional. The first survey identified participants to 
enroll in the study and provided information needed for the randomization, and the second survey provided implementation 
information on receipt and perception of the emails. This data source was used to create secondary outcome measures and 
address implementation questions. 

¢ OCCD email distribution data from the email platform Emma, which included information about the number of emails sent and 
received for each treatment group for each mailing, as well as opened emails, links clicked within emails, and requests from 
recipients to opt out of further emails. This data source was used to address the implementation research question. 

e Interviews conducted with key OCCD staff members in April 2020. This data source was used to gather information from key 
staff members involved in implementing the intervention and administering the surveys on treatment implementation (see 
appendix B for the semi-structured interview protocol). 


Sample. There were two populations of interest for this study. For the sign-up randomization (research question 1), eligible par- 
ticipants were early childhood education (ECE) workforce members at licensed facilities who were enrolled in the Oregon Registry 
Online but not signed up for the career lattice (10,716 individuals). Three-hundred and forty-eight individuals agreed to participate 
in the study, and the study team randomly assigned 50 percent of them at the individual level (174 individuals) to the treatment 
group (which received the signup emails) and 50 percent of them (174 individuals) to the control group (which received no emails). 
For the incentive randomization (research questions 2 and 3), eligible participants were Oregon Registry Online registrants who 
were at career lattice steps 7, 7.5, or 8 as of June 2018 (2,726 individuals). These individuals were invited to participate in the study 
and take a baseline survey. Two-hundred and forty-four participants agreed to participate and completed the baseline survey (see 
table B10 in appendix B for a comparison of the population of eligible workforce members and the study samples). The study team 
randomly assigned them at the workplace level to one of two separate treatment groups—monetary incentive (81 individuals) 
or automatic scholarship enrollment (81 individuals)—or the control group (82 individuals; see table B11 in appendix B for power 
calculations that indicate the number of participants needed to detect impact). 


Methodology. For both randomizations OCCD sent emails approximately every three months to the treatment group for a total 
of five emails sent to each treatment group participant (see figures B1 and B2 in appendix B for examples of the emails). 
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For research questions 1, 2, and 3 the study team estimated the impacts of sending emails at the individual level, regardless 
of whether they were received (called “intent-to-treat effects”), using regression analysis. The study team also estimated the 
impacts of the emails among those who received them (called “treatment-on-the-treated effects”) and among specific groups 
(defined by race/ethnicity, age, primary language, workplace role, education level, workplace type, urban or rural location, and 
distance to college). For research question 4 the study team summarized within each treatment group data from the email plat- 
form on the number of delivered emails and the number of participants who opted out of receiving further emails. The study team 
compared the percentage of respondents in the treatment groups for each intervention who were observed opening and clicking 
on email links, as well as the percentage of respondents in the treatment and control groups who reported on the follow-up 
survey that they had seen the emails. The study team also analyzed interview data about the email distribution processes and 
survey data about participants’ perceptions and knowledge of the emails, monetary incentives, and scholarships. See appendix B 
for more detail regarding the study methods. 


Limitations. Descriptive statistics indicate that the treatment and control groups were similar on several characteristics but were 
not representative of the overall ECE workforce (see tables B9 and B10 in appendix B); this population of study participants might 
also respond to email nudges differently. This finding limits the ability to generalize the study’s conclusions to the wider Oregon 
ECE workforce. Sample sizes for both randomizations were small, which limited the study team’s ability to detect impacts (see 
table B11). Statistical significance is reported at less than the .10 level, which is a looser standard than the typically used .05 level 
to allow for this limitation. The overall rates of survey attrition—respondents who took the baseline survey but did not complete 
the follow-up survey—were 61 percent for the sign-up randomization and 40 percent for the incentive randomization (see appen- 
dix B for more information about attrition). This high attrition affects the secondary outcomes; thus, research questions 1b, 2b, 
and 3b are exploratory analyses with limitations. Attrition limits the ability to use follow-up survey information in the analysis, 
particularly for the incentive randomization, which suffered from differential attrition—a difference between the proportion of 
the treatment group and the control group participants who did not complete the follow-up survey—as fewer control group par- 
ticipants responded to the survey than treatment group participants. Additionally, the study team did not interview educators to 
understand how the emails were perceived due to restrictions on the number of subjects who could be interviewed and the study 
timeline. Another important limitation is that the study timeline—originally planned for 16 months—might have been too short 
to detect impacts in some outcomes, particularly for scholarship recipients who might have needed more time to plan to attend 
college. This already-tight timeline was shortened by about four months due to a state policy change, which might have further 
reduced the ability to detect impacts. 


Findings 


This section presents results on the impact of the sign-up emails on registration for the career lattice and work- 
place retention (sign-up randomization); the impact of emails combined with a monetary incentive at earlier-than- 
usual steps in the career lattice on career lattice movement, professional development, and workplace retention 
(incentive randomization); and the impact of emails combined with automatic scholarship enrollment on schol- 
arship use, career lattice movement, college credit hours earned, and workplace retention (incentive randomiza- 
tion). In addition, this section includes results on reported motivation to take college coursework or complete a 
degree and identification as an early learning professional. The section concludes with implementation findings. 


Sending up to five emails encouraging career lattice sign-up had no detectable impact on career 
lattice sign-up or workplace retention 


Emails encouraging ECE workforce members who were not in the career lattice to sign up for it did not result in a 
detectable impact on sign-up or workplace retention. There were no statistically significant differences between 
the sign-up randomization treatment and control groups in the rate of sign-up or the rate of workplace retention 
during the study period (table 1). During the study period approximately 19 percent of the treatment group and 
15 percent of the control group signed up for the career lattice. Approximately 80 percent of the treatment group 
was retained at the same workplace compared with 81 percent of the control group; however, this difference was 
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Table 1. Sending the career lattice emails had no detectable impact on career lattice sign-up or workplace 
retention from 2018 to 2019 among study participants from the Oregon early childhood education workforce 


Treatment Control 
group mean group mean Estimated impact 
Outcome (standard deviation) (standard deviation) (standard error®) Effect size> Sample size‘ 
Signed up for the career lattice 0.186 0.154 0.031 0.086 289 
between 10/12/2018 and 12/31/2019 (0.390) (0.363) (0.294) 
Workplace retention 0.803 0.811 -0.008 -0.021 280 
(0.399) (0.393) (0.007) 


Note: None of the effects was statistically significant at p < .10. 

a. Adjusted using randomization inference tests. See appendix B for a description of how standard errors were estimated. 

b. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

c. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


not statistically significant. Results were similar for the treatment-on-the-treated analysis (see table C1 in appen- 
dix C). 


Sending emails had no detectable impact on career lattice sign-up or workplace retention for groups of partic- 
ipants defined by age range, primary language, education level, race/ethnicity, workplace role, workplace type, 
urban or rural location, or distance to the closest in-state public college or university (see table C2 in appendix C). 


The email encouraging career lattice sign-up had no detectable impact on identification as an early learning pro- 
fessional. The study team also explored whether receiving the emails encouraging career lattice sign-up increased 
ECE workforce members’ identification as early learning professionals, as reported in the OCCD survey data, and 
found no influence on this outcome (see table C3 in appendix C). About 18 percent of the treatment group iden- 
tified as a professional more strongly on the follow-up survey than on the baseline survey compared with 28 
percent of the control group. However, these results were not statistically significant. Because of the high non- 
response rate on the follow-up survey (61 percent), these results should be interpreted with caution. 


Sending up to five emails offering a monetary incentive at an earlier-than-usual step in the career 
lattice had a positive impact on training hours recorded but no detectable impact on career lattice 
movement, college credit hours earned, or workplace retention 


For ECE workforce members with at least the Child Development Associate credential and less than an associate 
degree (or equivalent training hours), providing access to a monetary incentive of up to $200 before step 9 of 
the career lattice through the incentive randomization had a statistically significant positive impact on training 
hours recorded but no detectable impact on career lattice movement, college credit hours earned, or workplace 
retention (table 2). On average, the treatment group had more community-based training hours than the control 
group (42.5 compared with 32.2), a statistically significant and meaningful difference. Treatment-on-the-treated 
estimates were also statistically significant for training hours recorded but not for other outcomes (see table C4 
in appendix C). 


Offering a monetary incentive at an earlier-than-usual step in the career lattice had a statistically significant pos- 
itive impact on total community-based training hours recorded and a statistically significant negative impact 
on workplace retention for some groups of treated participants. However, there were no statistically significant 
impacts by group on career lattice movement or college credit hours earned (see table C5 in appendix C). 
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Table 2. Sending monetary incentive emails had an impact on training hours recorded but not on career 
lattice movement, college credit hours earned, or workplace retention from 2018 to 2019 among study 
participants from the Oregon early childhood education workforce 


Treatment Control 

group mean group mean Estimated impact 
Outcome (Ge lreelaome(\Uraecela) Mm cieclater-TaeMe(-\r-latela) Mmm Cle-lalel-lae 1a ce) ao] Effect size> Sample size‘ 
Increased at least one step in the career 0.195 0.179 0.015 0.040 155 
lattice since 9/1/2018 (0.399) (0.386) (0.020) 
College credit hours earned between 15.143 18.968 -3.825 -0.050 155 
9/1/2018 and 12/31/2019 (46.806) (75.882) (7.701) 
Total community-based training hours 42.458 32.229 10.229t 0.343 155 
recorded between 9/1/2018 and 12/31/2019 (48.034) (29.797) (7.451) 
Workplace retention 0.813 0.855 -0.042 -0.118 151 

(0.392) (0.354) (0.331) 


t Significant at p < .10. 

Note: None of the effects was statistically significant at p < .05. 

a. Adjusted using randomization inference tests. See appendix B for a description of how standard errors were estimated. 

b. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

c. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Providing a monetary incentive at an earlier-than-usual step in the career lattice had no detectable impact on 
increase in identification as a professional, motivation to take a college course, or motivation to earn a degree. 
The emails offering small monetary incentives also had no detectable influence on four secondary outcomes: 
increase in identification as a professional, motivation to take a college course in the next year, motivation to 
take a college course in the next five years, and motivation to earn a degree in the next five years (see table 
C6 in appendix C).* Because of the high nonresponse rate on the follow-up survey (40 percent) and differential 
response rates between the treatment and control groups, these results should be interpreted with caution. 


Sending up to five emails about automatic scholarship enrollment had no detectable impact on 
scholarship use, career lattice movement, college credit hours earned, or workplace retention 


For the incentive randomization there were no statistically significant differences between the treatment and 
control groups in scholarship use (receipt of money), career lattice movement, college credit hours earned, or 
workplace retention during the study period (table 3). The treatment group that received emails about automatic 
scholarship enrollment had a higher rate of scholarship use (approximately 4 percentage points higher); however, 
this difference was not statistically significant. Overall rates of scholarship use were low among both treatment 
and control groups. Compared with the control group, the treatment group had a lower rate of career lattice move- 
ment (8 percentage points lower), earned three fewer college credit hours, and had a lower workplace retention 
rate, although these effects were also not statistically significant. Results were similar for the treatment-on-the- 
treated effect—that is, for those who opened and clicked ona link in at least one email (see table C7 in appendix C). 


The scholarship emails did not have a detectable impact on scholarship use, college credit hours earned, or work- 
place retention for groups of participants defined by age range, primary language, education level, race/ethnicity, 
workplace role, workplace type, urban or rural location, or distance to the closest in-state public college or uni- 
versity but did have a detectable negative impact on career lattice movement for some groups (see table C8 in 
appendix C). 


4. These outcomes were measured as whether respondents reported high or very high motivation. 
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Table 3. Sending scholarship emails had no detectable impact on scholarship use, career lattice movement, 
college credit hours earned, or retention from 2018 to 2019 among study participants from the Oregon early 
childhood education workforce 


Treatment Control 
group mean group mean Estimated impact 
Outcome (standard deviation) (standard deviation) (standard error?) Effect size> Sample size‘ 
Used a community college scholarship 0.063 0.026 0.037 0.232 158 
(0.244) (0.159) (0.092) 
Increased at least one step in the career 0.100 0.179 -0.079 -0.206 158 
lattice between 9/1/2018 and 12/31/2019 (0.302) (0.386) (0.080) 
College credit hours earned between 16.038 18.968 2.931 -0.039 158 
9/1/2018 and 12/31/2019 (41.755) (75.882) (4.593) 
Workplace retention 0.838 0.855 -0.018 —-0.050 156 
(0.371) (0.354) (0.027) 


Note: None of the effects was statistically significant at p < .10. 

a. Adjusted using randomization inference tests. See appendix B for a description of how standard errors were estimated. 

b. Calculated by dividing the estimated impact by the standard deviation of the outcome for the control group. 

c. Refers to the sample size for the regression analysis and includes both treatment and control group participants who had data for the outcome. 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Scholarship emails had a positive impact on motivation to take a college course in the next year and the next five 
years, as well as motivation to earn a degree in the next five years, but had no detectable impact on increase 
in identification as a professional. The scholarship emails had a statistically significant and meaningful positive 
impact on participants’ motivation to take a college course in the next year, to take a college course in the next 
five years, and to earn a degree in the next five years, with effect sizes ranging from moderate to large (0.38- 
0.68; see table C9 in appendix C). Because of the high nonresponse rate on the survey (40 percent) and dif- 
ferential response rates between the treatment and control groups for the incentive randomization, this result 
should be interpreted with caution. There was no detectable impact of the emails on increase in identification as 
a professional. 


After randomization the email campaigns were implemented as planned, with emails sent and 
delivered to all intended participants, although the interventions ended sooner than planned due to a 
state policy change 


OCCD staff members reported that all treatment group emails were successfully sent and delivered to intended 
email recipients, as confirmed by OCCD’s email marketing platform (that is, emails were reported received by 
the participant’s email account). Only seven emails sent to the treatment groups across the intervention period 
received an undeliverable notice when initially sent, and those were ultimately successfully delivered, with receipt 
confirmed via the email platform. Only 10 recipients across both trials asked to be taken off the mailing list, and 
the email platform automatically removed them from future mailings. This indicates that OCCD staff members 
were able to reach ECE workforce members through email as planned for the study. 


Most treatment group participants opened at least one email, but only a quarter to a third clicked on a link. Most 
study participants in the treatment groups opened at least one of the five emails sent during the study period: 84 
percent of participants in the sign-up randomization treatment group, 92 percent of participants in the monetary 
incentive randomization treatment group, and 95 percent of participants in the automatic scholarship enrollment 
randomization treatment group (table 4). However, these statistics might not reflect the extent to which partici- 
pants read the email content. 
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Table 4. Most treatment group participants from the Oregon early childhood education workforce opened at 
least one email in 2018 or 2019 (percent) 


Monetary incentive Automatic scholarship 
Sign up randomization randomization enrollment randomization 
treatment group treatment group treatment group 
Email tracking platform outcome (em 0) (n=77) | (ea:1°) 
Opened at least one email 84 92 95 
Opened and clicked on link in at least one email 26 34 25 


Note: n indicates the sample size of the specified treatment group. Treatment group participants received an initial email and four reminder emails for a 
total of five emails. 


Source: Authors’ analysis of Oregon Center for Career Development in Childhood Care and Education data from 2018 and 2019. 


Because engaging with the email is an important determinant of having successfully received the information on 
signing up for the career lattice, the monetary incentive, or scholarship enrollment, the study team also looked 
at the percentage of recipients who engaged by clicking on at least one link embedded in the email. For example, 
the sign-up email provided a link to the Oregon Registry Online website where participants could sign up. The 
incentive email included links to webpages, such as an online training calendar, a map of colleges, a question and 
answer page, and the Oregon Registry Online steps and education awards website. 


Only a quarter to a third of treatment group participants clicked on an email link, indicating that many who 
received an email did not take further action through the email (although they might have gone to the OCCD 
website directly). About 26 percent of participants in the sign-up emails treatment group, 34 percent of partic- 
ipants in the monetary incentive treatment group, and 25 percent of participants in the automatic scholarship 
enrollment treatment group clicked on a link in at least one email (see table 4). The study team conducted addi- 
tional analyses involving participants who opened and clicked on at least one email to examine the impact of 
receiving the email (that is, treatment-on-the-treated estimates; see tables C1, C4, and C7 in appendix C). There 
was no detectable impact on outcomes other than a positive impact of the monetary incentive treatment (incen- 
tive randomization) on training hours. 


Few control group members received information about the treatment. Survey and interview data suggest that few 
control group members received information about the treatment during the study, leaving little concern about 
this reducing the ability to find effects. According to OCCD administrators, no treatment-related emails were sent 
to any control group participants. In the follow-up survey fewer than 10 people across the control groups report- 
ed receiving a treatment email from OCCD. These control group participants might have mistakenly identified 
nonintervention emails from OCCD (such as the survey invitation itself) as treatment emails; alternatively, control 
group participants might have received a forwarded treatment email from a person in the treatment group. These 
findings indicate that instances of the control group receiving emails and inadvertently becoming aware of the 
treatment were minor, if they occurred at all. However, the follow-up surveys had low response rates (see table 
B12 in appendix B), so findings based on these surveys should be interpreted with caution. Moreover, interviews 
with OCCD staff members indicate that details of the treatment conditions were not communicated to outside 
entities to avoid the potential for discussing the special incentives with ineligible ECE workforce members. Like- 
wise, according to OCCD staff interviews, frontline OCCD staff members were not made aware of full treatment 
details to avoid accidentally discussing them with an ineligible workforce member. 


OCCD staff members perceived the process of coordinating the email campaigns as successful but also report- 
ed a need for more personalized communications in future campaigns. Overall, OCCD staff members reported 
in interviews that they believed the interventions were straightforward to administer, the awards and scholar- 
ships were easy to administer, and the interventions did not increase staff burden due to the successfully imple- 
mented administration processes and systems they had established from the beginning (that is, using existing 
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administrative data, database systems, and email systems). Staff members reported being able to clearly delin- 
eate in their data systems which special monetary award or scholarship incentive an individual was eligible for 
as part of study participation. Therefore, when study participants interacted with the OCCD staff, staff members 
readily knew which award amount participants could be offered (depending on treatment status) or if partici- 
pants had been automatically enrolled in the scholarship program. Staff members reported no issues with cor- 
rectly administering the special awards and scholarship process. Part of this success was attributed to clear and 
regular communication with frontline staff members, who were able to follow the outlined plan when interacting 
with ECE workforce members. 


On further reflection during interviews after the email campaigns concluded, OCCD staff members noted that 
the emails could have been better designed to elicit opening and clicking for further information—for example, 
by including more personalized information. They also reported wanting to explore whether participants were 
familiar with the OCCD logo and viewed it as a trusted source of information or whether the emails should include 
the logo of other affiliated organizations. 


Small numbers of survey respondents in the treatment groups reported implementation challenges, such as not 
receiving the emails or encountering barriers to signing up for the career lattice, applying for an education award, 
or accessing the scholarship. Small numbers (ranging from 8 to 16) of survey respondents in the treatment groups 
indicated that they did not receive the emails or encountered barriers to signing up for the career lattice, applying 
for an education award, or accessing the scholarship, while others (ranging from 8 to 14) said the emails motivat- 
ed behavior change (these sample sizes are very small in some cases and should be interpreted with caution). 


To examine how the email interventions were implemented and perceived, the study team reviewed the follow- 
up survey responses among treatment group respondents to provide additional context. Some respondents indi- 
cated that they were motivated to change their behavior by the email information, although those impacts were 
not detected statistically. Very small numbers (ranging from 5 to 8) of survey respondents reported not signing up 
for the career lattice because they were unsure how to do so or did not know about the lattice, suggesting that 
some treatment group participants might not have read the emails. Some (ranging from 10 to 12) respondents 
also said that they did not apply because they did not feel that they met the requirements for an education award 
or they did not access the scholarship because they lacked time to pursue college education. Moreover, only a 
third to half of survey respondents from the treatment group reported receiving a treatment email (see table C10 
in appendix C). Although based on small sample sizes, this suggests that some participants did not engage with 
the treatment emails as intended, which could help explain the null findings of the study. (See tables C11—C14 in 
appendix C for additional data on email receipt and motivation to continue professional development.) 


A key implementation challenge was that both the sign-up and incentive interventions ended earlier than planned 
because of a state policy change. The unanticipated shortening of the study period by four months exacerbat- 
ed concerns regarding an original timeframe that was shorter than ideal. The original timeframe of less than 18 
months was already short, and OCCD staff noted in interviews that this type of intervention would ideally be 
fielded for at least two years to have an opportunity to show impact. The limited timeframe made it unlikely 
that the study would find an impact on participants’ progress through certain career lattice steps or plans for 
returning to college, both of which can take multiple years. About 7 percent of all educators at step 7, 7.5, or 8 at 
the start of the study, regardless of whether they were participating in the study, moved to a higher step in the 
planned 16-month study period. Historically, OCCD data show that the median time to move from one of these 
three steps to a higher step is at least 400 days (about 13 months). Moving from step 7 to 7.5 could be achieved 
by completing as few as two courses, which is potentially feasible for a full-time worker in less than one year 
by taking one course per quarter. Moving from step 7.5 to 8 or from step 8 to 8.5 requires taking at least three 
courses or 10 credit hours, which might be more challenging within the study timeframe for a workforce member 
who is working full time. 
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Implications 


This study found that emails, alone or coupled with information about monetary incentives, induced changes in 
professional development attainment but were not sufficient to induce changes in career lattice sign-up or schol- 
arship use among participating Oregon ECE workforce members, at least in the short term. These findings suggest 
the following implications for research and practice. 


Research implications 


First, Oregon could consider using qualitative or survey research to better understand barriers to enrolling and 
progressing in the career lattice. The study team did not explore the potential difficulties ECE workforce members 
face in submitting the required applications to sign up for the career lattice, to move up a step in the career 
lattice, to receive a monetary incentive tied to a step, or to receive a scholarship. OCCD staff members are avail- 
able to help with the application process, but workforce members might not be aware of that. More research is 
needed to understand whether these applications are a barrier and, if so, how that barrier could be mitigated. 


Second, more research is needed to explain why some ECE workforce members choose to increase their edu- 
cation despite low compensation in the field and how shifting education requirements and other state policy 
changes influence these decisions. Exploring why some ECE professionals choose to increase their education 
could shed light on ways to provide incentives to others and could enable a better understanding of how compen- 
sation levels and state policies interact with this decision. 


Third, future impact studies could investigate whether more intensive behavioral nudges or behavioral nudges 
delivered through alternative methods (such as text messages or social media) would encourage ECE work- 
force members to increase their education and training. The positive finding related to training hours, as well 
as the small number of survey respondents who indicated that these interventions motivated them to continue 
their education and training, suggests that emails are a promising strategy that needs additional study to fully 
understand how they could be leveraged. However, for the most part the study findings suggest that light-touch 
behavioral nudges might not be enough to induce changes in career lattice sign-up, continuing postsecondary 
education, and scholarship use. This suggests that a different approach, such as more frequent messages or use 
of text messages in lieu of email, might be needed to encourage those outcomes. Given the frequent imple- 
mentation of low-cost interventions such as email or text messages both nationally and in Oregon, policymakers 
will likely need more information about how different types of behavioral nudges can encourage ECE workforce 
members to increase their education levels. 


Fourth, future research could also examine whether monetary incentives or scholarships alone are effective in 
Oregon and at what levels of financial support. This study did not test whether monetary incentives alone are 
effective or whether a scholarship alone is effective. 


Finally, longer implementation and study periods might be needed to test education outcomes. Some of the out- 
comes examined in the study, such as increased course credits, require at least one college term to be com- 
pleted, and a longer implementation and study period might be needed to test the effectiveness of an email 
campaign. The impact of signing up for the career lattice could have been detected in the study timeframe of one 
year. However, to earn training hours or college credits, recipients need to plan for attending classes, sign up for 
classes, attend the classes, and then provide evidence that classes were completed before OCCD can move the 
ECE workforce member up the career lattice. Each of these steps takes time, and the outcomes in this study might 
have been measured too early to capture results. It is encouraging that the follow-up survey data indicate that 
some treatment group participants were motivated to enroll in college courses. A study period of two or three 
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years might show significant results, and future research related to these outcomes should consider a longer 
timeframe than that of the current study. 


Practice implications 


First, the study findings (such as the low rate of career lattice sign-up) from the three interventions suggest a 
need for better messaging and information about the workforce registry, career lattice, and available financial 
incentives and scholarships. For example, advertising the career lattice in different ways to reach different ECE 
workforce groups and testing these different forms of messaging could mitigate the workforce’s lack of informa- 
tion. In addition, such messaging could clearly describe the benefits of participation in the career lattice and of 
additional professional development to help motivate the ECE workforce on a personal level. 


Second, the findings also suggest that additional supports, such as release time, could be offered to help ECE 
workforce members pursue higher education and overcome barriers such as a lack of time to participate in pro- 
fessional development. A small number of ECE professionals in the study reported lacking time to take courses to 
move up a step or take advantage of the scholarship. This indicates that this workforce might face barriers related 
to available time to participate in education, perhaps because of work and family obligations. 


Third, future email campaigns could use different messaging techniques to increase the probability of the work- 
force receiving the information. Emails could be used for additional purposes, including research, and might 
benefit from simplified language, personalized messages, and translation into recipients’ primary language. This 
study demonstrates the power of email messaging for research and practice among Oregon ECE professionals, 
given that the interventions were implemented as planned and that OCCD reported that most treatment group 
participants received the emails. OCCD had rarely used email to communicate with the ECE workforce prior to 
this study. OCCD might consider using emails to communicate with the workforce or engage the workforce with 
other research studies. 


However, the study findings suggest that participants might not have absorbed the information in the emails. For 

example, many treatment group participants did not click on the links, and a few treatment group participants 

indicated that they were unaware or unsure of how to sign up for the career lattice despite data indicating that 

the emails had been delivered to them. Participants’ perceptions could be due to such factors as readability, 

interest in the content, or distrust in the emails (that is, fear that it might be a hoax or scam), suggesting that the 

quality of the emails could be improved in future campaigns. Based on the finding from interviews with OCCD 

staff members that the emails could be better designed to elicit a stronger response, the study team suggests 

considering the following changes for future email campaigns: 

e Simplify email language where possible and connect content to topical ECE issues and current events. 

e Personalize emails—for example, including information about the recipient’s current step level and what 
coursework or training hours are needed to move to the next step. 

e Ensure that emails are available in a recipient’s primary language (expanding beyond English and Spanish). 

e When possible, craft email campaigns to use more widely recognizable logos and known organization names to 
increase trust in the content. 


Additionally, OCCD might want to consider email alternatives, such as text messages or social media, in communi- 


cating with the ECE workforce. These alternative methods might be more effective at reaching this population or 
at reaching particular segments of the workforce. 
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